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LANE RECOGNITION APPARATUS FOR VEHICLE 
BACKGROUND OF THE INVENTION 

The present invention relates to an apparatus for 
recognizing a lane on which a vehicle or other mover 
travels . 

Various lane recognition apparatuses have been 
proposed in order to assist a driver to drive a vehicle 
along a lane or to automatically drive a vehicle. 
Japanese Patent Provisional Publication No. 11-259639 
discloses a lane recognition apparatus which is capable 
of determining whether each of the detected lane marker 
candidate point may be treated as a correct candidate 
poin^ on the basis of the dispersion of lane marker 
candidate points and an estimated line. 
SUMMARY OF THE INVENTION 

However, such a lane recognition apparatus yet has 
problems to be improved in order to further accurately 
and stably estimate a road shape. 

It is therefore an object of the present invention 
to provide an improved lane recognition apparatus which 
is capable of eliminating an erroneous recognition caused 
by an erroneous detection of lane marker candidate points 

An aspect of the present invention resides in a lane 
recognition apparatus for a vehicle, which comprises an 
image picking-up section picking up a road image ahead of 
the vehicle; a lane-marker detecting section detecting a 
plurality of lane -marker candidate points on the road 
image; a road model parameter calculating section 
calculating a road model parameter representative of a 
road shape ahead of the vehicle on the basis of the 
lane-marker candidate points; a deviation calculating 
section calculating a deviation between the lane marker 
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candidate point and an estimated candidate point 
estimated from the road model parameter; and a lane 
marker detection feasibility determining section 
determining whether the road image is employed for 
5 detecting the lane marker, on the basis of the deviation. 

Another aspect of the present invention resides in a 
method for recognizing a lane ahead of a vehicle, which 
method comprises a step for picking up a road image ahead 
of the vehicle; a step for detecting a plurality of lane 

10 marker candidate points from the road image; a step for 
calculating a road model parameter representative of a 
road shape ahead of the vehicle on the basis of the lane 
marker candidate points; a step for calculating a 
deviation between the lane marker candidate point and an 

15 estimated candidate point estimated from the road model 
parameter; and a step for determining whether the road 
image is employed for detecting the lane marker, on the 
basis of the detecting deviation. 

Further another aspect of the present invention 

20 resides in a lane recognition apparatus for a vehicle 

which comprises a camera and a processor. The camera is 
installed to the vehicle and picks up a road image ahead 
of the vehicle. The processor is coupled to the camera 
and is arranged to detect a plurality of lane marker 

25 candidate points from the road image, to calculate a road 
model parameter representative of a road shape ahead of 
the vehicle on the basis of the lane marker candidate 
points, to calculate a deviation between the lane marker 
candidate point and an estimated candidate point 

30 estimated from the road model parameter, and to determine 
whether the road image is employed for detecting the lane 
marker, on the basis of the deviation. 
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The other objects and features of this invention 
will become understood from the following description 
with reference to the accompanying drawings . 
BRIEF DESCRIPTION OF THE DRAWINGS 
5 Fig. 1 is a block diagram showing a lane recognition 

apparatus which is common in first to third embodiments 
according to the present invention. 

Fig. 2 is a flowchart showing a processing procedure 
of the lane recognition apparatus of the first embodiment. 
10 Fig. 3 is a flowchart showing a processing procedure 

of the lane recognition apparatus of a second embodiment. 

Fig. 4 is a flowchart showing a processing procedure 
of the lane recognition apparatus of a third embodiment . 

Fig. 5 is a view showing a relationship between a 
15 road coordinate system and a plane coordinate system 
employed in the present invention. 

Figs . 6A and 6B are views showing a road model 
according to the present invention. 

Fig. 7 is an explanatory view of lane marker 
20 detection small areas employed in the present invention. 

Fig. 8 is an explanatory view of a lamer marker 
detection result and a lane marker candidate point . 

Fig. 9 is a view showing a relationship between the 
lane marker candidate points and an estimated road model. 
25 Fig. 10 is an explanatory view of a deviation 

between the estimated candidate point and the lane marker 
candidate point. 

Fig. 11 is a graph showing a detectable threshold 
between detectable data and undetectable data. 
30 Figs. 12A is a view showing an example of a lane 

marker detectable image, and Fig. 12B is a view showing 
an example of a lane marker undetectable image. 



P01NM-082/00-00682 



DETAILED DESCRIPTION OF THE INVENTION 

Embodiments of the invention will now be described 
based on the drawings . 
[First Embodiment] 
5 Fig. 1 is a block diagram showing a first 

embodiment of a lane recognition apparatus 1 for a 
vehicle according to the present invention. Lane 
recognition apparatus 1 comprises a CCD camera 101 for 
picking up an image of a scene of a road ahead of the 

10 vehicle, a pre-process section 102 for uniformly 

processing an entire screen of a video signal from CCD 
camera 101, a lane marker detection small area setting 
section 103 for setting a plurality of small areas for 
detecting a lane marker on an input screen, a lane marker 

15 detecting section 104 for detecting a part of the lane 
marker in each small area as a lane marker candidate 
point, a road parameter calculating section 105 for 
calculating road parameters for representing the shape of 
the road ahead of the vehicle based on the result of lane 

20 marker detection executed at lane marker detecting 

section 104, a deviation calculating section 106 for 
calculating a deviation between the lane marker candidate 
point and the estimated candidate point estimated from 
the road model parameter, and a marker detection 

25 feasibility determining section 107 for determining 

whether or not the lane maker can be detected from the 
image on the basis of deviation calculating section 106. 
CCD camera 101 functions as an image pickup section of 
the invention. 

30 The lane recognition apparatus 1 comprises a 

processor 100 which is constituted by a microcomputer and 
which is coupled to CCD camera 101. The microcomputer is 
arranged to store the sections 101 through 107 of the 
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lane recognition apparatus in the form of software and to 
perform the processes of the sections 101 through 107 by 
executing a program control in processor 100. 

Now, there will be described steps for recognizing 
5 a lane executed by lane recognition apparatus 1 according 
to the present invention. 

Fig. 2 is a flowchart showing a flow of processes 
in the first embodiment according to the present 
invention. First, lane recognition apparatus 1 is 

10 started when a switch is operated by the driver or the 
vehicle is ignited at step S201. At step S202 road 
parameters for recognizing the shape of a road ahead of 
the vehicle are initialized. 

The road parameters are defined according to the 

15 following procedure. As shown in Fig. 5, a road 

coordinate system is defined as an XYZ system in which 
the center of an image pickup lens of CCD camera 101 is 
the origin. The X-axis extends from the right to the 
left as viewed in the traveling direction of the vehicle. 

20 The Y-axis extends upward in the direction of the height 
of the vehicle. The Z-axis is the optical axis of the 
lens along the traveling direction of the vehicle. As 
shown in Fig. 5, a plane coordinate system of an image 
processing screen is defined in which the origin is at 

2 5 the upper left corner of the screen. The x-axis 

horizontally extends, and the y-axis vertically extends 
from the top to the bottom in accordance with the 
direction in which a screen is scanned in television 
communication systems such as NTSC . 

30 The relationship between those two coordinate 

systems is as shown in Fig. 5 and, for simplicity, 
coordinate transformation from the road coordinate system 
to the plane coordinate system is expressed by Equations 
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(1) and (2) shown below where the origin of the plane 
coordinate system is located on the Z-axis of the road 
coordinate system as shown in Fig. 5. 

x = -fX/Z ---(1) 

Y = -fY/Z ---(2) 

where f represents a parameter that represents the focal 
length of the lens . 

While a planar structure of a road is defined by 
straight lines, curves having constant curvatures, and 
clothoid curves having constant curvature change rates 
for connecting the straight line and the curve, a section 
of the road of several tens meters ahead of a vehicle can 
be regarded as a curved road having a constant curvature 
or a straight road. Then, the shape of a lane marker was 
formulated as shown in Fig. 6A. Similarly, a 
longitudinal structure of the same was formulated as 
shown in Fig. 6B because it can be regarded as having a 
constant gradient. The formulae are respectively given 
as Equations (3) and (4) below. 

X = £- Z 2 + <f)Z + y c - iW ( 3 ) 

Y = rjZ - h. ---(4) 

where p represents the curvature of the road; 0 represents 
a yaw angle of the vehicle to the road; y c represents 
lateral displacement of the vehicle from a left lane 
marker; W represents the vehicle width; i represents the 
left line marker when it is 0 and represents a right lane 
marker when it is 1; jj represents a pitch angle of the 
optical axis of the lens to the road surface; and h 
represents the height of the camera above the ground. 

The shape of a lane marker projected on the plane 
coordinate system of the image processing screen can be 
formulated from Equations (1) through (4). Equations (1) 
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through (4) can be rearranged by eliminating X, Y, and Z 
to obtain the following Equations (5) through (10). 

x = (a + ie) (y - d) — + c - - - ( 5 ) 

y - d 

where a = -y c /h, (6) 

b = -f 2 hp/h, ---(7) 
c = -£<p + Co, ---(8) 

d = -f rj + d 0 , and ( 9 ) 

e = W/h. (10) 

The terms c 0 and d 0 are correction values that are 
required because the actual origin is located at the 
upper left corner of the image processing screen in spite 
of the fact that the origin of the plane coordinate 
system is on the Z-axis in the road coordinate system in 
Fig. 5. 

It is apparent from the above that the curvature of 
the road, the pitch angle and the yaw angle of the 
vehicle, and the lateral displacement of the vehicle 
within the lane can be estimated by identifying values of 
parameters a through e in Equation (5) which are 
satisfied by a lane marker candidate point detected 
through the image processing. 

Returning to Fig. 2, at step S203, an image signal 
of an image picked up by CCD camera 101 is inputted to 
pre-processing section 102 which performs a pre-process 
on the same. For example, as a pre-process for detecting 
a lane marker, primary space differentiation is performed 
with a Sobel Filter to emphasize a boundary between the 
lane marker and a road surface. The object of lane 
marker detection is such a boundary. Since a lane marker 
has regions which are nearly horizontal and regions which 
are nearly vertical depending on the curvature of the 
road, two edge images are created using horizontal 
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differentiation and vertical differentiation. Other edge 
emphasizing filters may be used to emphasize a boundary 
between the lane marker and road surface. Further, any 
method for emphasizing the boundary may be employed for 
5 characteristically extracting the lane marker. 

At step S204, a plurality of small areas are set in 
order to detect a lane marker which indicates the lane in 
which the vehicle is traveling on the screen. As shown 
in Fig. 7, the detection areas are set as predetermined 

10 sections defined along the shape of the road identified 

from the results of the previous image processing or road 
parameters which have been initially set . Since road 
model parameter calculating section 105 comprises a 
storage device for storing the road model parameter, at 

15 this step S204 processor 100 reads the road model 

parameter stored in the storage device and uses in the 
small area setting process. 

At step S205, processor 100 detects a straight line, 
which most likely seems to be a boundary between the lane 

20 marker and the road surface in the region in the small 
areas set in step S204, and defines the point on the 
straight line as a lane marker candidate point. As shown 
in Fig. 8, the upper-most point of the results of the 
straight line detection is defined as the lane marker 

25 candidate point. Each lane marker point is stored in the 
form of the coordinate value in the x-y coordinate system. 

At step S206, processor 100 calculates a deviation 
of each lane marker candidate point detected at step S205 
relative to the road model estimated from the result of 

30 the previous lane marker recognition result, as shown in 
Fig. 9. By defining such that a point, which is located 
on the road model and on the same y-coordinate of the 
lane marker candidate point treated as a calculated 
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object, is an estimated candidate point and that a 
x- coordinate of the lane marker candidate point is qi and 
a x-coordinate of the estimated candidate point is ri as 
shown in Fig. 10, the deviation pi is obtained by 
5 Equation ( 11 ) . 

pi = |qi - ri| ---(11) 
Since the deviation at each lane marker candidate 
point detected can be calculated by Equation (11), an 
average of the respective deviations pi is represented by 
10 Equation (12) and is used as a deviation p of the lane 
marker detection result in the present image, 
p = (total of all pi) 

/(number of lane marker candidate points) 

---(12) 

15 Equation (13) may be employed for the calculation 

of each deviation pi as follows . 

pi = (qi - ri) 2 ---(13) 
The model parameter is also read from the storage 
device of the road parameter calculating section 105 and 

20 used in this deviation calculating process. 

At step S207, on the basis of the deviation p 
calculated at step S206, it is determined whether or not 
it is possible to detect the lane marker from the 
inputted image. That is, marker detection feasibility 

25 determining section 107 of processor 100 determines 

whether the road image can be employed for detecting the 
lane marker. When a condition that the value of the 
average deviation p obtained from Equation (12) is 
greater than or equal to a predetermined value, is 

30 continuously maintained for a predetermined time period, 
it is determined that it is not possible to detect the 
lane marker from the input image. Further, when filtered 
data, which is obtained by treating time-series data of 
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the average deviation p using a low pass filter, is 
greater than a predetermined detectable threshold as 
shown in Fig. 11, it may be determined that it is 
impossible to detect the lane marker from the input image. 
5 Figs. 12A and 12B show input images which is determined 
whether the lane marker is detected from the input images 
by using the method shown in Fig. 11. Fig. 12A shows an 
input image from which the lane marker can be detected, 
and Fig. 12B shows an input image from which the lane 

10 marker cannot be detected. That is, when a film of 

rainwater is formed on a road surface during raining at 
night, reflection of roadway lighting or water stayed in 
trails of vehicles is picked up like as a lane marker, 
and therefore the erroneous detection is executed as 

15 shown in Fig. 12B. In Fig. 12B, the reflection of 

roadway lighting is picked up as vertical lines in a left 
hand side image , and the water stayed in trails is 
picked up as a right hand side. Accordingly, Fig. 12B 
shows a condition that the detection result is erroneous, 

20 since these reflection and water are picked up like as a 
lane marker. 

When it is determined at step S207 that it is not 
possible to detect the lane marker from the input image, 
the program returns to step S202 and repeats the 

2 5 above-discussed processing. When it is determined at 

step S207 that it is possible to detect the lane marker 
from the input image, the program proceeds to step S208. 

At step S208, processor 100 calculates parameters 
for the road model from the positions of the lane marker 

30 candidate points on the screen obtained at step S205. At 
this time, processor 100 reads the road parameter stored 
in the storage device of road parameter calculating 
section 105. An extended Kalman filter may be used as 
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means for estimating an equation for a two-dimensional 
road model based on a result of detection of a lane 
marker candidate point through image processing. 

Equation (14) is derived from the above-described 
Equations (1) through (4). Equation (14) is used as an 
output equation in configuring the extended Kalman filter, 
and the value of an x-coordinate at a y-coordinate value 
defined on a plane under image processing is calculated 
from the road curvature and a state quantity of the 
vehicle. 

, Y c . W. . , f 2 hp 

x = (--£- + x-)(Y + f v ) + £ f0 ---(14) 

h h 2(y + f 77) 

The estimated state quantity in the extended Kalman 
filter includes lateral displacement y c of the vehicle, 
road curvature p, vehicle yaw angle <p of the vehicle, 
pitch angle rj of the vehicle, and height h of CCD camera 
101. Further, the focal length f of the lens and the 
lane width W are treated as constant values. A state 
equation expressed as Equation (15) shown below is 
obtained if it is defined as a random walk model in a 
discrete system which is driven by white Gaussian noises 
v on the assumption that a change of each estimated state 
quantity behaves stochastically. 

"1 0 0 0 0"|[y c l fl 0 0 0 01[v yc " 
01000/? 0 1 0 0 0 v p 

00100 (p + 00100 (15) 

00010?7 00010 v 7 
OOOOlh 00001 v h 
When State Equation (15) and Output Equation (14) 
are simplified into Equations (16) and (17) shown below, 
the extended Kalman filter is configured by Equations 
(18) through (21). 

x s (k+l) = A s x s (k) + G s v(k) (16) 



y c 




p 




V 




h 
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x = g(x s ,y) ---(17) 
x s (k + 1 | k) = A s x s (k | k - 1) + K(k) frOO - g(x s (k |k-l) f5 (k))} 

---(18) 

K(k) = AsPfkJ^CCPCkJ^+R)- 1 ---(19) 
P(k+1) = A s M(k)A s T + GsQG s T ---(20) 
M(k) = P(k){I-C T (CP(k)C T +R)- 1 CP(k)} ---(21) 
where M(k) represents covariance of states which is 
determined by the process; P(k) represents covariance of 
later states obtained from observed values; K(k) 
represents the gain of the Kalman filter; Q represents 
covariance of observation noises; R represents covariance 
of process noises; I represents a unit matrix; k 

represents a lag; and C . MM^J^J . 

dx s 

Although in the above embodiment the height h of 
CCD camera 101 is estimated on the basis of the premise 
that the width W of the lane is constant, it will be 
understood that the width of the lane W may be estimated 
on the basis of the premise that the height h of CCD 
camera 101 is constant. Further, although the first 
embodiment is arranged to calculate the road parameter by 
using the Kalman filter, the present invention is not 
limited to this embodiment and may be arranged to 
calculate the road parameter by using the other 
estimating method or identifying method such as the least 
square method. 

Returning to Fig. 2, at step S209, processor 100 
stores the new road parameter calculated at step S208 in 
the data storage region of the storage device of road 
model parameter calculating section 105. The stored road 
parameter is used in the next image processing including 
the setting of the lane marker detection areas, the 
calculation of the deviation, and the calculation of the 
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new road parameter. Thereafter, the routine returns to 
step S203 wherein a new image for the next processing is 
picked up. 
[Second Embodiment] 
5 The lane recognition apparatus 1 of a second 

embodiment comprises basic elements as shown in Fig. 1, 
as is similar to the first embodiment. 

Fig. 3 is a flow chart showing a flow of processes 
in the second embodiment according to the present 

10 invention. First, lane recognition apparatus 1 is 

started when a switch is operated by the driver or the 
vehicle is ignited at step S301. Next, at step S302 road 
parameters for recognizing the shape of a road ahead of 
the vehicle are initialized, and a content of a counter 

15 CNTR is reset (CNTR=0). The definition of the road 

parameter in the second embodiment is the same as that in 
the first embodiment. 

Steps S303 through S307 correspond to steps S203 
through S207 of the first embodiment. More specifically, 

20 at step S303 pre-processing section 102 of processor 100 
receives the signal of the signal of the image picked up 
by CCD camera 101, and executes the pre-processing. At 
step S304 processor 100 sets the small areas in order to 
detect a lane marker which indicates the lane in which 

25 the vehicle is traveling on the screen. At step S305, 

processor 100 detects a straight line, which most likely 
seems to be a boundary between the lane marker and the 
road surface in the region in the small areas set in step 
S304, and determines the point on the straight line is 

30 determined as a lane marker candidate point. Further, at 
step S306 processor 100 calculates a deviation of the 
lane marker candidate point detected at step S305 
relative to the road model estimated from the result of 
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the previous lane marker recognition result. Furthermore, 
at step S307, processor 100 determines whether or not it 
is possible to detect the lane marker from the inputted 
image on the basis of the average deviation p calculated 
5 at step S306. That is, marker detection feasibility 
determining section 107 of processor 100 determines 
whether or not the road image can be employed for 
detecting the lane marker. 

When it is determined at step S307 that it is 

10 possible to detect the lane marker from the inputted 
image, the routine proceeds to step S308 wherein the 
content of counter CNTR is set at zero (CNTR=0). 

At step S309 following the execution of step S308, 
processor 100 calculates parameters of the road model 

15 from the position of the lane marker candidate points on 
the image obtained at step S305 as is similar to the 
processing of step S208 of the first embodiment. Further, 
at step S310, processor 100 updates the road parameter, 
that is, stores the new road parameter calculated at step 

20 S309 in the data storage region of the storage device of 
road model parameter calculating section 105 as is 
similar to the processing of step S209 of the first 
embodiment . 

When the determination at step S307 is negative, 
25 that is, when it is determined that it is not possible to 
detect the lane marker from the inputted image, the 
program jumps to step S311 wherein the content of counter 
CNTR is incremented by 1 ( CNTR+1-^CNTR ) . 

At step S312 following the execution of step S311, 
30 it is determined whether or not the content of counter 
CNTR is smaller than a predetermined value N. When the 
determination at step S312 is affirmative (CNTR<N), the 
program returns to step S303 wherein the next image 
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processing is executed without updating the road 
parameter. When the determination at step S312 is 
negative (CNTRsrN) , the program returns to step S302 
wherein the initialization of the road parameter and the 
5 reset of counter CNTR are executed again. 
[Third Embodiment] 

The lane recognition apparatus 1 of a third 
embodiment comprises basic elements as shown in Fig. 1, 
as is similar to the first embodiment. 

10 Fig. 4 is a flowchart showing a flow of processes 

in the third embodiment according to the present 
invention. First, lane recognition apparatus 1 is 
started when a switch is operated by the driver or the 
vehicle is ignited at step S401. Next, at step S402 road 

15 parameters for recognizing the shape of a road ahead of 
the vehicle are initialized, and a content of a counter 
CNTR is reset (CNTR=0). The definition of the road 
parameter in the third embodiment is the same as that in 
the first embodiment . 

20 Steps S403 through S407 are basically similar to 

steps S203 through S207 of the first embodiment. More 
specifically, at step S403 pre-processing section 102 of 
processor 100 receives the signal of the signal of the 
image picked up by CCD camera 101, and executes the 

25 pre-processing is executed. At step S404, processor 100 
sets the small areas in order to detect a lane marker 
which indicates the lane in which the vehicle is 
traveling on the screen. At step S405, processor 100 
detects a straight line, which most likely seems to be a 

30 boundary between the lane marker and the road surface in 
the region in the small areas set in step S404, and 
determines the point on the straight line is determined 
as a lane marker candidate point. Further, at step S406 
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processor 100 calculates a deviation of the lane marker 
candidate point detected at step S405 relative to the 
road model estimated from the result of the previous lane 
marker recognition result. Furthermore, at step S40 7, 
5 processor 100 determines whether or not it is possible to 
detect the lane marker from the inputted image on the 
basis of the deviation p calculated at step S406. 

When it is determined at step S407 that it is 
possible to detect the lane marker from the inputted 

10 image, the routine proceeds to step S408 wherein the 
content of counter CNTR is set at zero (CNTR=0). 

At step S409 following the execution of step S408, 
processor 100 calculates parameters of the road model 
from the position of the lane marker candidate points on 

15 the image as is similar to the processing of step S208 of 
the first embodiment. Further, at step S410, processor 
100 stores the new road parameter calculated at step S409 
in the data storage region of the storage device of road 
model parameter calculating section 105 as is similar to 

20 the processing of step S209 of the first embodiment. 

When the determination at step S407 is negative, 
that is, when it is determined that it is not possible to 
detect the lane marker from the inputted image, the 
program jumps to step S411 wherein the content of counter 

25 CNTR is incremented by 1 (CNTR+1-^CNTR) . 

At step S412, it is determined whether or not the 
content of counter CNTR is greater than a predetermined 
value N. When the determination at step S412 is 
affirmative (CNTR>N) , the program proceeds to step S413 

30 wherein the road parameter is estimated using a 

sequential-type estimator such as the Kalman filter and 
on the basis of the changing state of the road parameter 
until the previous processing without using the present 
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parameter candidate point. When the determination at 
step S412 is negative (CNTRsN) , the program returns to 
step S402 wherein the initialization of the road 
parameter and the reset of counter CNTR are executed 
5 again . 

With the thus arranged embodiments according to the 
present invention, it becomes possible to eliminate wrong 
recognition caused by the incorrect detection of the lane 
marker candidate points. Further, it becomes possible to 

10 improve the reliability of the lane marker detection 
result to the diagnosis of the wrong recognition. 
Furthermore, it is possible to convert the offset 
quantity of the pixels of the image into the offset 
quantity on the road surface. The embodiments according 

15 to the present invention is arranged such that when the 
image is determined that it is not proper to employ for 
detecting the lane marker, the detection result of the 
lane marker candidate points obtained from the improper 
image are not used to calculate the road model parameter. 

20 Accordingly, this arrangement prevents the road shape of 
the road ahead of the host vehicle from being incorrectly 
detected . 

This application is based on a prior Japanese Patent 
Application No. 2000-397747. The entire contents of a 
25 Japanese Patent Application No. 2000-397747 with a filing 
date of December 27, 2000 are hereby incorporated by 
reference . 

Although the invention has been described above by 
reference to certain embodiments of the invention, the 
30 invention is not limited to the embodiments described 

above. Modifications and variations of the embodiments 
described above will occur to those skilled in the art in 
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light of the above teachings. The scope of the invention 
is defined with reference to the following claims. 



